Systematic reviews: time to address clinical and policy relevance as well as methodological rigor.

Well As Methodological Rigor Compared with other study designs, well-done randomized trials provide the most valid estimate of the benefits of health interventions because they minimize bias. Systematic reviews of randomized trials identify all studies that have addressed a particular question, and metaanalyses combine the results by using methods that minimize bias (1). Many consider systematic reviews to be the best source of information for making clinical and policy decisions. Such groups as the Cochrane Collaboration have established standards for the conduct and reporting of systematic reviews (2). In this issue, Shojania and colleagues (3) extend this tradition of subjecting research methods to close scrutiny. They describe how quickly the conclusions of 100 systematic reviews published in ACP Journal Club became out of date because of newly completed randomized trials that changed either the statistical significance or the magnitude of the summary treatment effect when they were added to the original review. Fifty percent of reviews were out of date within 5.5 years after publication and 23% were out of date within 2 years. Not surprisingly, the results were most likely to change for cardiovascular interventions (presumably because of the large number of trials in cardiovascular medicine) or if the trials in the original review were heterogeneous. Unfortunately, Shojania and colleagues’ findings do not allow authors to accurately identify the time at which their specific systematic review will be out of date. Because outdated reviews can be misleading, we believe that researchers should update a literature search annually to identify studies that might change the results of the original review, unless a search of clinical trial registries assures them that no randomized trials addressing the topic of the review are ongoing. Because the updating search should use the same search strategy as the original review, updating should take relatively little time. In the past 2 decades, studies like that of Shojania and colleagues have improved the methodological quality of systematic reviews. We suggest that the time has come to devote similar energy to increasing the likelihood that the content and format of reviews are useful to a variety of decision makers. Despite advances in the conduct and reporting of systematic reviews, current evidence suggests that they are used less frequently by clinicians and policymakers than one might think. Patients use them even less frequently. A systematic review of the information-seeking behavior of physicians found that textbooks (many of which do not rely on evidence from systematic reviews) are still the most frequently consulted source of information, followed by advice from colleagues (4). Similarly, nurses and other health professionals seem to refer to evidence from systematic reviews infrequently in decision making (5, 6). Analyses of selected policymaking processes in Canada and at the World Health Organization found that evidence from systematic reviews was used infrequently (7, 8). Given that systematic reviews of randomized trials are less susceptible to bias than the opinions of experts and observational data, why do people fail to use them when making policy, clinical, or personal decisions? In trying to answer these questions, we consider the clinical relevance of the questions addressed by the systematic reviews, the format of systematic reviews, and the failure of authors to place their findings in a clinical context. Most systematic reviews address highly specific questions that interest the author. Authors seldom consult with policymakers, clinicians, patients, or health care managers about what’s important to them. Clinicians often struggle with broad questions, such as, “What’s the most effective treatment for insomnia?”, rather than the narrow questions that researchers prefer, such as, “What’s the evidence that benzodiazepines are effective for the treatment of insomnia?” Also, many important clinical questions are not addressed by systematic reviews. For example, a review of the Cochrane Collaboration databases for “dizziness” and “syncope” yielded no completed reviews of the management or investigation of these common symptoms. Systematic reviews are entirely dependent on the relevance of the randomized trials that have been conducted. Except for relatively rare pragmatic trials, the trials that most systematic reviews comprise involve highly selected patients who receive care from highly selected physicians, which in some circumstances raises important questions about their generalizability. (Actually, we feel that people sometimes worry too much about generalizablity. Clinical trials will never be done in every conceivable group of patients; therefore, clinicians must learn the art of applicability—that is, asking, “Is our patient really so different from patients included in the trials that we can’t apply the results?”). Randomized trials generally underreport adverse events (9) and often do not study patients for a sufficient period to detect important side effects. These shortcomings can make them, and therefore systematic reviews, insufficient to drive clinical practice on their own. Systematic reviews are often off-putting to clinical readers. To meet the criteria established for the reporting of systematic reviews, most reviews are lengthy (Cochrane reviews are often longer than 30 pages), appear complicated to those not trained in systematic reviews, and take a long time to read and appraise. Thus, it is not surprising that clinicians use them infrequently and prefer more userfriendly formats for accessing evidence. Authors of systematic reviews tend to focus on docuAnnals of Internal Medicine Editorial